The present invention relates to a method and apparatus for automatically replacing billboards in a vide image.
The present invention has particular use in electronic replacement of billboards in a stadium or other venue but can be used to provide accurate data relating to camera orientation for other purposes.
In previous systems it has been proposed to electronically replace billboards in a stadium which are viewed by a viewer on television. The billboards in the stadium are televised by a TV camera and the boards are electronically altered so that the TV viewer at home sees a different board to the spectator in the stadium or other venue.
The known systems such as those described in U.S. Pat. No. 5,266,933, an apparatus and method for electronically altering video images is disclosed. The apparatus and method disclosed in the U.S. patent and also in U.S. Pat. No. 5,353,392 whilst theoretically allowing replacement of billboards do not solve the many practical problems encountered in real environments. Most of these problems are related to the recognition and replacement processes.
Relying entirely on pattern recognition techniques which utilize only the video signal to identify and localise billboards for replacement introduces major problems which affect the practical value of such a system.
Clearly, any pattern recognition scheme, including those described in U.S. Pat. No. 5,264,933 and U.S. Pat. No. 5,353,392 must rely on useful visible features in the image that can be compared with pre-defined descriptions. Such features should be located inside the billboard or at its neighbourhood.
In realistic situations, the visibility of these features might change, continuously or otherwise from practically zero to a some threshold visibility which allows the pattern recognition scheme to work properly. These changes can occur in the direction of growing or reducing visibility.
Such situations include:
Acceleration or de-acceleration of camera motion introducing a huge amount of blur.
Excessive zooming-in or zooming-out of the billboard.
Excessive occlusion by players.
Entering or exiting a camera""s field of view by any combination of pan, tilt and zoom operations.
Any combination of the above mentioned mechanisms.
Therefore, in practical situations, a continuous replacement of billboards, is not possible. Even if an interrupted replacement was allowed, it would require a delay of at least a few seconds to decide whether the resulting replacement interval is acceptable or not. Such a delay is usually not permitted in live broadcasting of sports events.
Replacing arbitrary billboards introduces further problems. A seamless replacement requires to identify the foreground objects occluding the billboard in order to inhibit replacement at places of occlusion. Foreground objects mainly consist of players but also the ball or other objects. Consider now a player with a red shirt, occluding a part of a similarly red portion of a billboard. Colour contrast cannot be used robustly to identify occlusion. Furthermore, since the player is a non-rigid object, motion or shape information cannot be used accurately enough to guarantee perfect replacement.
Another problem which may arise in practical situation is resolution of billboard identity. Consider two identical billboards positioned at two different locations in the arena. Suppose different replacement billboards are assigned to each of these physical billboards, then one must be able to tell which one is which. This can prove to be extremely difficult especially if no unambiguous features are visible.
This invention described a robust system foe billboard replacement, based on some or all of the following key elements:
Pan, tilt, zoom and focus sensors attached to the camera, which enable after a proper set up procedure to estimate the presence and location of billboards in any given video field.
Image processing methods and their embodiment which enable to refine the sensors"" estimates.
Physical billboards which are coloured properly to enable the efficient detection of occlusion by chroma-key techniques.
Colour variation or a pattern within the physical billboard for further enhancing the performance of the image processing methods.
The present invention has a first object to provide a method and apparatus which enables identification of the location of a billboard or other static object in a stadium or other venue in any weather conditions with any panning speed of the camera and with any other change in camera parameters.
The present invention therefore provides apparatus for automatic electronic replacement of a billboard in a video image including an automatic camera orientation measurement apparatus including motion measurement means operative to measure the Field of View (FOV) of the TV camera relative to a known reference position.
The present invention also preferably provides apparatus for automatic electronic replacement of a billboard in a video image, including image processing means for processing video signals generated by the TV camera, in which said processing means includes calibration means for periodically automatically calibrating the motion measurement means, apparatus in which the motion measurement means includes means for measuring the pan tilt, zoom or focus of the camera relative to known reference positions and apparatus in which the motion measurement means includes means for measuring the pan tilt, zoom and focus of the camera relative to known reference positions.
The present invention therefore uses dynamic recalibration to correct for residual sensor errors or abberations in an imperfect model and for sensor drift over time. Thus it is possible in accordance with the present invention to use less stable sensors and the apparatus and method in accordance with the present invention can accommodate movement in the camera position. The image correction process for calibration of the sensors eliminates the necessity to keep the sensors stable by mechanical means by recalibration automatically with reference to the video image.
In the initial set up procedure corrections can be incorporated for calibration for billboards which are, for example, not in the centre of the Field of Viewxe2x80x94for example a billboard which is in the top left-hand corner of the screen can be adjusted by, for example, 3 pixels to take into account abberations in the camera.
Further problems which arise in the prior art systems are firstly when the billboard is either substantially totally occluded or secondly, is occluded by an object, such as a player, of the same colour as the real sign on the billboard.
This can firstly, as explained above, lead to non-recognition of the billboard and also secondly, to difficulty in satisfactory replacement of the billboard.
In the first case the real billboard can have already been replaced in the video image but if the camera zooms into a close up or if a different camera is used for the close up then the lock may be lost due to only a very small portion of the billboard being in view. In the second case the player may have on a strip which is the same colour as the billboard. The prior art systems propose to distinguish the billboard from the player on the basis of movement if the colours are the same and to analyse the xe2x80x9cmovingxe2x80x9d pixels to determine occlusion. This is reasonable in theory but fails in practice since not all payers are moving at all times. Thus, if a number of players move in front of a billboard and one player remains after the others have moved on, the electronics will not be able to distinguish on movement grounds. Since colours are distorted by floodlighting, shadows, differences in reflectivity and different lighting conditions for a foreground player and a background billboard, there will be occasions in practice where the system fails. In such cases either the original billboard may re-appear on the video image or the replacement billboard will not be accurately occluded.
It is again possible to introduce a delay in the video transmission to enable the electronic signal processing to be more accurate but this does not solve the practical problem where a plurality of players move in different directions to occlude a billboard. The necessary delay is considered unacceptable and will in any case not solve all of the above problems.
In accordance with a preferred embodiment of the present invention it is proposed to replace the real billboards with chroma-key panels or with delineated areas forming chroma-key panels.
Chroma-key is essentially an occlusion technique allowing, for example, a news reader to stand and move about in front of a chroma-key board, usually coloured blue or another suitable colour. The news reader (foreground) is distinguished from the chroma-key board (background) by colour differentiation and can thus move in front of the replacement background with normal occlusion of the foreground and background. This technique is very well known in television studio systems and is described in numerous US patents, including U.S. Pat. Nos. 2,974,190 and 4,200,980.
Recently, several systems which combine camera sensors with chroma-key for the purpose of coordinating the movement of the graphics backgrounds with those of the camera, have been described and demonstrated.
(Ref. K. Haseba et al., Real-timing compositing system-of a real camera image and a computer graphic image, International Broadcasting Convention, Sep. 16-20 1994, Conference publication No. 397, IEE 1994, pp. 656-660).
In principle, such an arrangement could be used for billboard replacement where the sensors unambiguously solve the recognition problem and the chroma-key billboard helps to handle occlusion properly. However due to some major differences, this arrangement should be enhanced. These enhancements are the basis of the present invention.
In a virtual set application, the camera is typically 2-10 metres away from the foreground and the entire field of view is usually replaced. In comparison, a billboard may be several hundred metres from the camera and therefore a replacement system using sensors is much more susceptible to sensor errors:
Due to the large focal distances, the same sensor accuracy will translate to larger geometric registration errors.
Consider a rotary encoder of 81000 pluses/revolution, then the angular precision is 0.0044 degrees or 75 micro-radians. The repeatability is twice as bad. Consider a shooting range of 100 m with a field of view of 4 metres, then the FOV is 40 milli-radian. The error translates to 768*? 150/20000=2.88 pixels.
Since the field of view includes many stationary objects (including billboards) which are not replaced, the human observer will be much more sensitive to the registration errors. Additional errors may originate from lens distortion, rotation axis which does not pass through the focal point, non-zero roll angle, etc.
Chroma-key is basically a technique for studios where the illumination is carefully designed and controlled and the controls of the chroma-keyer are carefully adjusted for the specific arrangement of blue-screen colour and illumination.
In a sports event, the conditions may be highly non-ideal and require some modification to the chroma-key algorithms. In particular, the keyer parameters should be adapted to the specific billboards being replaced due to changes in illumination across the arena.
Accordingly in the present invention it is proposed to use chroma-key panels and to replace these in the video image by the replacement billboards.
Since it is necessary for perfect occlusion that the players or other occluding objects are of different colour to the chroma-key panels, it is proposed in a further preferred embodiment to provide chroma-key panels in which the colour of the panel may be changed, for example, by using a rotating billboard structure which is known in the art. One side could, for example, be blue and another green. Green may be preferable in a sports environment since players tend not to wear green as this would not contrast with the background sports surface.
In a further preferred embodiment and in particular where a plurality of billboards require replacement, a patterned chroma-key board is used. The pattern may be of any suitable shape but is preferably selected to be suitable for the size and shape of the billboard or series of billboards and also to the anticipated video conditions. Thus if a billboard is only able to be viewed from a long distance then a different pattern will be selected to a billboard which is to be viewed in close up
The pattern may comprise different colours or may be different shades of the same colour. The pattern may comprise vertical and horizontal lines or may comprise a decorative pattern, a discernible advertisement, company logo or other suitable wording which may be more aesthetically acceptable.
The use of a pattern allows further discrimination of the position of the camera and may allow movement of the camera from a fixed position.
The camera orientation data can be transmitted together with the video signal and will identify the position of the billboard in any weather, lighting or occlusion conditions. No reference is required to any feature within the sports venue to identify the position of the billboard.
The camera sensors can be accurate to a few pixels or in physical terms to approximately 1 cm at a range of about 100 metres thereby enabling accurate replacement of any billboard. The recalibration can be carried out continuously or only periodically, particularly if an initial adjustment of the calibration of billboards not in the centre of the FOV is recorded on set up.
By use of the chroma-key techniques there is no requirement to transmit any occlusion data since this can be readily inserted at a receiver and the occlusion inserted in the normal manner.
In a preferred arrangement within a stadium or other sports venue real billboards with normal advertising material will be situated on one side of the stadium to be viewed by a first plurality of cameras and chroma-key billboards will be situated on another or the opposite side to be viewed by a second plurality of cameras. This, for example, the home nation may view the normal billboards, with the international TV audience seeing only substituted boards.
The present invention also provides a method for electronically replacing a billboard in a video image display, generated by a camera, comprising the steps of:
a. identification of the position of a rectangular billboard in a stadium or other venue, said identification step comprising specifying on the video display the billboard to be replaced by identification of its four corners at a first camera position;
b. storing the identification information;
c. monitoring the movement of the camera in pan, tilt and zoom;
d. storing the monitored movement of the camera on a field by field basis; and
e. analysing the size and position of the billboard to be replaced from the information recorded in its first known position and the stored movements of the camera to provide information relating to the size, perspective and position of the billboard in the present video field;
f. storing in a billboard replacement store a replacement billboard to be used in replacement of the billboard in the stadium;
g. electronically altering the size and perspective of the replacement billboard in accordance with the camera motion information to conform to the size and perspective of the billboard to be replaced in the present video frame; and
h. electronically replacing the billboard in the present video frame by the replacement billboard.
In a preferred embodiment the step of analysing the size and position of the billboard to be replaced comprises a further step of analysing a plurality of video scan lines to provide fine adjustment information relating to the exact size, perspective and position of the billboard to be replaced.
In a further preferred embodiment the billboard to be replaced is blank and is of colour suitable for chroma-key replacement. Such colour may be of a blue or green shade, due to the fact that these colours are rarely found in human skin and hair.
In a still further preferred embodiment the chroma-key billboard is patterned with a pattern of a suitable shape for the purpose of facilitating the fine adjustment process mentioned above. The step of analysis of the size and position of the billboard comprises the analysis of the pattern to ascertain the exact position of the billboard.
In a further embodiment the correction of the sensor-based prediction by the analysis of the pattern will be controlled by a figure of merit (accuracy estimate) for the analysis, which will be computed automatically.
In a further embodiment the step of electronically replacing the billboard in the present video field by the replacement billboard include the step of superimposing occluding objects by use of the chroma-key techniques.
In a further preferred embodiment the billboard to be replaced can be changed to best match the colours and shades of colours on the players costumes, for the purpose of providing a good contrast between the billboards and the players. For example, if these costumes contain shades of blue, then a green billboard may be selected.
The backing colour can be selected between blue, green and red. In order for the chroma-keyer to calculate all parameters necessary to perform proper image compositing, the system requires a sample of the background colour as reference. This step can be done automatically by scanning the image and detecting the purest and brightest colour. Advanced chroma-keyers enable the user to manually select the area to be sampled.
In a further preferred embodiment the chroma-key apparatus will have a multiplicity of set up conditions, each corresponding to a different region of the stadium. The camera pan, tilt and zoom information will allow to load the corresponding set up conditions.
In a further embodiment, the fine adjustment information will be used to compensate drift errors of the sensors. In a practical situations, the sensor error will have a significant portion which is at temporal frequencies which are much lower than the video field rate. Thus these sensor-induced errors can be reliably estimated from good video field and subtracted from subsequent measurement.
The present invention also provides apparatus for carrying out the method of electronically replacing the billboard as specified hereinbefore.